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DETECTING HALFTONE MODULATIONS EMBEDDED IN AN 

IMAGE 



CROSS-REFERENCE TO RELATED APPLICATIONS 

This application is related to U.S. Application Serial No. 09/594,235, filed June 
5 14, 2000, by Jan P. Allebach et al., which is incorporated herein by reference. 

TECHNICAL FIELD 

This invention relates to systems and methods of detecting halftone 
modulations embedded in an image. 

BACKGROUND 

10 Digital halftoning (or spatial dithering) is the method of rendering the illusion 

of continuous tone pictures in an arrangement of binary picture elements. In the 
case of rendering a continuous tone image with a digital output device, such as an 
ink jet printer or a laser printer, digital halftoning involves simulating the continuous 
tone image with groups or cells of dots. 

15 In general halftoning involves generating a binary, or two-tone, image from a 

continuous tone, or contone or grayscale, image. Grayscale images are continuous 
tone black (or some other single color) and white images, whereas contone images 
may be either full color images or single-color and white images. In either case, a 
halftone image is generated from a contone (full color or grayscale) image using any 

20 one of a variety of halftoning techniques, including threshold arrays or dithering 

(e.g., clustered dots, dispersed dots and stochastic screens), adaptive processes (e.g., 
error diffusion), and interactive processes (e.g., least squares and direct binary 
search) . 

Many different methods for embedding information into an image also have 

25 been proposed. 

For example, bar coding is a well-known category of document or image 
marking techniques for densely encoding digital information in a small image space 
without regard to how visible the encoded information is to a human viewer. A bar 
code symbol is a pattern of parallel bars and spaces of various widths that represent 
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data elements or characters. The bars represent strings of binary ones and the 
spaces represent strings of binary zeros. A conventional "one-dimensional" bar code 
symbol contains a series of bars and spaces that vary only in a single dimension. 
One-dimensional bar code symbols have relatively small information storage 
capacities. "Two-dimensional" bar codes have been developed to meet the 
increasing need for machine-readable symbols that contain more information than 
one-dimensional bar code symbols. The information storage capacity of two- 
dimensional bar code symbols is increased relative to one-dimensional bar codes by 
varying the bar code patterns in two dimensions. Common two-dimensional bar 
code standards include PDF417, Code 1, and Maxicode. One-dimensional and two- 
dimensional bar code symbols typically are read by optical scanning techniques (e.g., 
by mechanically scanned laser beams or by self-scanning charge-coupled devices 
(CCD's)) that convert a printed bar code symbol into electrical signals. The electrical 
signals are digitized and decoded to recover the data encoded in the printed bar code 
symbol. 

Data glyph technology is another category of information embedding 
techniques that is particularly advantageous for use in image applications that 
require a high density rate of embedded data and require the embedded data to be 
robust with respect to decoding. Data glyph technology encodes digital information 
in the form of binary l's and O's that are then rendered in the form of distinguishable 
shaped marks such as very small linear marks. Generally, each small mark 
represents a digit of binary data, and the linear orientation of the particular mark 
determines whether the particular digit is a digital 1 or 0. 

Other document or image marking techniques have been proposed for 
embedding information in an image so that the information substantially is 
imperceptible to a human viewer (i.e., in a manner that simultaneously minimizes 
image distortion caused by embedding the information) while permitting reliable 
decoding of the information. For example, many different digital watermarking 
techniques have been proposed. In general, a digital watermark is designed to 
produce an embedded signal that is imperceptible to a human viewer so as not to 



diminish the commercial quality and value of the image being watermarked, while 
producing an embedded signal that is resistant to tampering. 

In another approach, U.S. Patent No. 6,141,441 discloses a technique for 
decoding message data that has been encoded into a printed color image as a series 
of small image regions (referred to as "signal cells") that carry the encoded message. 
Each signal cell is composed of a spatial pattern of colored subregions that 
collectively have an overall average color. The colors of the subregions are defined 
as changes (modulations) to the average color in one or more directions in a multi- 
dimensional color space. The decoding technique uses a set of valid signal blocks, 
each of which is a unique pattern of color modulated subregions. There is a valid 
signal block for each valid message value defined in the coding scheme. The 
decoding operation first locates the positions of the signal cells in the acquired image 
and then subtracts the local average color of each signal cell from the cell to produce 
a received signal block. The decoding operation determines the respective valid 
signal block corresponding to each of the received signal blocks by comparing each 
valid signal block to each received signal block. One implementation of the decoding 
technique decodes signal cells that have been arranged in the acquired image in a 2D 
array by synchronizing an imaginary grid-like structure with the most likely position 
of all of the signal cells. In one embodiment, a color space direction is selected for 
the color modulations that results in the differently colored subregions of a signal cell 
being substantially imperceptible to a human viewer, thus making the pattern that 
carries the message substantially imperceptible in an encoded image. 
Still other information embedding schemes have been proposed. 

SUMMARY 

The invention features systems and methods of detecting halftone 
modulations that are embedded in an image. The halftone modulations represent an 
ordered sequence of digital halftone microscreens that encodes information 
embedded in the image. The invention provides an effective, robust and efficient 
scheme for detecting such halftone modulations without knowledge of the original 
image before halftone modulation. 



In one aspect of the invention, information embedded in an image is detected 
by applying a respective filter to the image for each of two or more different halftone 
modulations to identify an ordered sequence of halftone modulations embedded in 
the image. 

5 Embodiments of the invention may include one or more of the following 

features. 

Each halftone modulation preferably is generated from a respective dither 
matrix. Each filter preferably corresponds to a matched filter for a respective 
halftone modulation. 

10 In some embodiments, the step of applying a respective filter to the image 

comprises convolving the filter with the image. A score map preferably is generated 
for each halftone modulation based upon the convolution of the corresponding filter 
S and the image. Halftone modulations embedded in regions of the image may be 
£j identified based upon the generated score maps. One halftone modulation preferably 
5 is is selected as corresponding to an image region to which multiple halftone 
S modulations are identified based upon the generated score maps. Halftone 

* modulations may be identified in the image based upon a first threshold applied to 
□ the score maps. The threshold may be lowered in regions to which no halftone 

* modulation has been identified based upon the first threshold. In addition, image 
ffi 20 tone may be re-mapped in image regions to which no halftone modulation has been 
2 identified based upon a lowered threshold. 

The image may comprise a plurality of image levels, and filters may be 
applied to a single image level at a time. For example, the image may be a grayscale 
image comprising multiple gray levels and filters initially may be applied to a middle 
25 gray level of the image (e.g., a 50% gray level of the image). Filters may be applied 
to a different gray level of the image to resolve uncertainties or ambiguities, or both. 

A system for detecting information embedded in an image and a computer 
program for detecting information embedded in an image also are described. 

Other features and advantages of the invention will become apparent from the 
30 following description, including the drawings and the claims. 



4 



DESCRIPTION OF DRAWINGS 

FIG. 1A is a block diagram of an encoder producing from an original image a 
halftone image containing information encoded as an ordered sequence of halftone 
microscreens and a printer producing a hard copy of the encoded halftone image. 
5 FIG. IB is a block diagram of a scanner producing a scanned image from an 

encoded halftone image and a decoder extracting from the scanned image an ordered 
sequence of halftone microscreens embedded in the encoded halftone image. 

FIG. 2 is a diagrammatic representation of the encoder of FIG. 1A, including a 
sequence generator, a composite halftone screen generator, and a halftone module. 
10 FIG. 3 is a flow diagram of a method of encoding information in a halftone 

image as an ordered sequence of halftone microscreens. 

FIG. 4A is a diagrammatic representation of a dither matrix from which a set 
of grayscale halftone microscreens may be generated. 

FIG. 4B is a diagrammatic representation of the dither matrix of FIG. 4A with 
15 the entries corresponding to a 10% gray level circled. 

FIG. 4C is a diagrammatic representation of a halftone microscreen generated 
from the dither matrix of FIG. 4A for a 10% gray level. 

FIG. 5 is a diagrammatic representation of the decoder of FIG. IB, including a 
correlator and a sequence recovery module. 
20 FIG. 6 is a flow diagram of a method of decoding information embedded in a 

halftone image as an ordered sequence of halftone microscreens. 

FIG. 7 is a diagrammatic representation of a matched filter for a halftone 
microscreen generated from the dither matrix of FIG. 4A for a 10% gray level. 

FIG. 8 is a diagrammatic representation of a score map generated by 
25 convolving a sharpened version of the matched filter of FIG. 7 with an encoded 
halftone image. 

FIG. 9A is a diagrammatic representation of an encoded halftone image. 
FIG. 9B is a diagrammatic representation of the encoded halftone image of 
FIG. 9 A with regions identified as corresponding to a first embedded halftone 
30 microscreen marked by shading. 
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FIG. 10 is a flow diagram of a method of resolving ambiguities in the decoding 
method of FIG. 6. 

FIG. 11 is a flow diagram of a method of resolving uncertainties in the 
decoding method of FIG. 6. 
5 FIG. 12 is a diagrammatic representation of the encoded halftone image of 

FIG. 9 A with regions corresponding to first and second embedded halftone 
microscreens marked by shading after ambiguities and uncertainties have been 
resolved in accordance with the methods of FIGS. 10 and 11, respectively. 

DETAILED DESCRIPTION 

10 In the following description, like reference numbers are used to identify like 

elements. Furthermore, the drawings are intended to illustrate major features of 
exemplary embodiments in a diagrammatic manner. The drawings are not intended 
to depict every feature of actual embodiments nor relative dimensions of the depicted 
elements, and are not drawn to scale. 

15 Referring to FIG. 1A, in one embodiment, an encoding system 10 includes an 

encoder 12 that is configured to encode information 14 as an ordered sequence of 
two or more different halftone microscreens in an original image 16 to produce an 
encoded halftone image 18. Encoding system 10 also includes a printer 20 that is 
configured to produce a hard copy 22 of encoded halftone image 18. Encoder 12 

20 may be a computer or other programmable processor, and printer 20 may be a 
conventional printer (e.g., a LaserJet® printer available from Hewlett-Packard 
Company of Palo Alto, California, U.S.A.). Original image 16 may be a continuous 
tone grayscale image or a continuous tone color image. The ordered sequence of 
halftone microscreens may be embedded in one or more levels (i.e., one or more 

25 gray levels or color concentration levels) of original image 16. The resulting halftone 
modulations in encoded halftone image 18 are substantially imperceptible to a 
viewer; however, the halftone modulations may be detected readily and reliably by 
an appropriately configured decoding system. 

Referring to FIG. IB, in one embodiment, a decoding system 30 includes a 

30 scanner 32 that is configured to produce a continuous tone scanned image 34 from 
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hard copy 22 of encoded halftone image 18 by a conventional inverse-halftoning 
process. Decoding system 30 also includes a decoder 36 that is configured to detect 
the halftone modulations encoded in hard copy 22 of halftone image 18 and, thereby, 
extract information 38 encoded in the detected ordered halftone modulation 

5 sequence. Scanner 32 may be a conventional desktop optical scanner (e.g., a 

ScanJet® scanner available from Hewlett-Packard Company of Palo Alto, California, 
U.S.A.) or a portable scanner (e.g., a CapShare® portable scanner available from 
Hewlett-Packard Company of Palo Alto, California, U.S.A.), and decoder 36 may be a 
computer or other programmable processor. 

io As described in detail below, in accordance with the above-described 

encoding and detection scheme, the embedded halftone modulations may be 
detected efficiently, effectively and robustly without knowledge of original image 16 
to provide a truly viable blind information embedding and decoding approach. 
Referring to FIG. 2, in one embodiment, encoder 12 includes a sequence 

15 generator 40, a composite halftone screen generator 42, and a halftone module 44. 
Sequence generator 40 is configured to generate an ordered sequence of halftone 
microscreens from multiple sets of microscreens 46 based upon the information 14 to 
be encoded in original image 16. Composite halftone screen generator 42 is 
configured to generate composite (or mosaic) halftone screens 48 from the ordered 

20 sequence of halftone microscreens received from sequence generator 40. Halftone 
module 44 is configured to generate encoded halftone image 18 from the composite 
halftone screens 48 received from composite halftone screen generator 42 by a 
halftone screening process. As explained in detail below, each microscreen set 46 
includes a microscreen for each level (1 through N) of original image 16. 

25 Accordingly, together sequence generator 40 and composite halftone screen 

generator 42 are configured to generate a composite halftone screen for each level of 
original image 16. 

Referring to FIGS. 3, 4A, 4B and 4C, in operation, encoder 12 is configured to 
produce from original image 16, information 14 and microscreen sets 46 an encoded 
30 halftone image 18 as follows. 
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Initially, a set of two or more different dither matrices is constructed (step 50). 
The dither matrices preferably are designed to tile together seamlessly and may be 
constructed as described in co-pending application Serial No. 09/594,235, filed June 
14, 2000, by Jan P. Allebach et al. As shown in FIG. 4A, each dither matrix 52 

5 includes an arrangement of histogram numbers respectively corresponding to the 
levels of original image 16. In this embodiment, original image 16 is assumed to be a 
grayscale image with 255 levels (1 through 255). 

Next, sets of microscreens 46 are generated from the dither matrices (step 54). 
The histogram numbers of each dither matrix 52 specify the number and 

10 arrangement of dots to add at each microscreen image level. For example, as shown 
in FIGS. 4B and 4C, histogram numbers greater than 230 are marked (by ovals 56) to 
indicate where dots 57 should be placed to produce a 10% gray level microscreen 58. 
Microscreens for other image levels may be generated in a similar way. 

Sequence generator 40 then generates a sequence of microscreen sets 46 based 

15 on the information 14 to be encoded for each image level (step 60). Since the dither 
matrices from which the microscreens are generated are different, the microscreens 
may be tiled in an ordered microscreen sequence that encodes information 14. In 
general, the information 14 to be encoded includes a string of data items each 
indicating one of a set of valid values in a coding scheme. The data items are not 

20 restricted in any way as to the nature of the information to be conveyed and may, for 
example, represent character symbols in a language using ASCII or UNICODE 
character encoding, or the compressed or encrypted form of such symbols. The data 
items may indicate data in the form of "0" and "1" symbols, or may indicate a set of 
values that define another coding scheme. 

25 Optionally, the microscreen sequences may be encoded with an error 

correction code (step 62). In some embodiments, the microscreen sequences also 
may incorporate other schemes (e.g., a redundancy scheme) to facilitate decoding. 

Additional details regarding the process of constructing the sets of 
microscreens 46 may be obtained from the above-mentioned co-pending application 

30 Serial No. 09/594,235, filed June 14, 2000, by Jan P. Allebach et al. 
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Referring to FIG. 5, in one embodiment, decoder 30 includes a correlator 70 
and a sequence recovery module 72. Correlator 70 is configured to correlate regions 
of the continuous tone scanned image 34 with a corresponding microscreen of one of 
the microscreen sets 46 to produce a correlation image 74 for selected image levels. 

5 Sequence recovery module 72 is configured to recover the microscreen sequence 
from each correlation image 74 to extract information 38. 

Referring to FIGS. 6, 7, 8, 9A and 9B, in one embodiment, decoder 30 is 
configured to extract information 38 from scanned image 34 as follows. 

Initially, correlator 70 generates one matched filter for each microscreen set 46 

10 for a selected image level (step 80). It has been discovered that, in many cases, a 
substantial fraction of the microscreens may be identified in scanned image 34 using 
matched filters designed for an image level of 50% (e.g., a 50% gray level). As 
shown in FIG. 7, in this embodiment, the resulting matched filters 82 each includes a 
matrix of l's at locations where dots should be placed to produce the microscreen 

15 corresponding to the selected image level, and -l's at the remaining locations. For 
example, the matched filter of FIG. 7 corresponds to the 10% gray level microscreen 
58 of FIG. 4C, and includes l's at the dot locations of microscreen 58 and -l's at the 
remaining locations. 

Next, each matched filter is sharpened (step 84). The matched filters may be 

20 sharpened by a conventional sharpening filter. For example, the following 

sharpening filter may be applied to each matched filter to improve the ability of 
correlator 70 to identify microscreens in scanned image 34. 
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25 The resulting sharpened matched filters are convolved with scanned image 34 

to generate a score map for each of the microscreens that have been generated for 
the selected image level (step 86). As shown in FIG. 8, each score map 88 includes 
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clusters 90 (or pinnacles) of relatively light pixels surrounded by a relatively dark 
background region 92. Each pinnacle 90 in a given score map 88 corresponds to a 
central portion of a region of scanned image 34 where there is a strong likelihood 
that the corresponding microscreen has been embedded. In the embodiment of FIG. 

5 8, pinnacles appear to be arranged in a regular array corresponding to the central 
portion of each tile location across scanned image 34, indicating that the same 
microscreen has been embedded at each tile location. In other embodiments, 
different microscreens may be embedded in scanned image 34, in which case 
pinnacles would appear in a given score map only at the locations where the 

10 corresponding microscreen has been embedded. 

In some embodiments, scanned image 34 may be dewarped to remove 
scanner distortion before the sharpened matched filters are convolved with scanned 
image 34 (step 86). For example, scanner distortion may be dewarped with a 
mapping T: RxR -> RxR from the coordinates of the true image (I) to the coordinates 

15 of the scanned image (F) such that: 

I(x, y) = I*(x\ y') = r(T x (x, y), T y (x, y)) 

The mapping may be approximated by polynomials with coefficients that are 
20 obtained by a conventional least squares approximation. For some embodiments, 
however, the scanner distortion is minimal and scanned image 34 need not be 
dewarped. 

Next, the centroids of the pinnacle locations are identified in each score map 
(step 94). In one embodiment, pinnacle locations are identified by applying a 

25 threshold to each score map to determine the coordinates of regions likely to contain 
the centers of the tile locations. The coordinates of each pinnacle region are adjusted 
with respect to a modulus corresponding to the tile size (e.g., the coordinates would 
be computed based on mod 32 for tiles with dimensions of 32x32). The resulting 
coordinates are tallied and the coordinate with the highest count is selected as the 

30 center of periodicity for the score map being analyzed. The pinnacle regions of each 
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score map are coalesced into the selected center of each tile location and outliers are 
removed. 

The preliminary identification results are recorded in a final score map for 
each microscreen (step 96). As shown in FIGS. 9A and 9B, in one embodiment, a 

5 final score map 98 for a given microscreen may consist of a series of regions 100 that 
are marked over a scanned image 102 corresponding to tile locations where the 
microscreen has been identified. The remaining, unmarked tile locations correspond 
to image regions where the microscreen was not found, at least in the first pass. 
Next, ambiguities and uncertainties in the recorded preliminary results are 

10 resolved (step 104). Ambiguities arise when two or more microscreens are identified 
at the same tile location. Uncertainties arise when none of the microscreens can be 
identified at a particular tile location (e.g., when none of the microscreen score maps 
registers a hit). 

Referring to FIG. 10, in one embodiment, ambiguities may be resolved by a 

15 probabilistic analysis as follows. Initially, the score maps for the different 

microscreens that have been identified at an ambiguous tile location are compared 
(step 110). If one of the identified microscreens has a dominant score (i.e., a higher 
score map pinnacle value produced by the convolution of the corresponding 
sharpened matched filter and the scanned image) (step 112), that microscreen is 

20 identified with the ambiguous tile location (step 114). In some embodiments, the 
dominant microscreen must have a score that is greater than the scores of the other 
identified microscreens by a selected threshold - which may be determined 
empirically based upon the background noise level - before being identified with the 
ambiguous tile location. In other embodiments, however, the dominant microscreen 

25 merely has to have a score that is marginally greater than the scores of the other 
identified microscreens in order to be identified with the ambiguous tile location. If 
there is no dominant microscreen (step 112), the process of FIG. 6 is repeated for a 
different image level (step 116). The selected matched filter image level may be 
greater than 50% if the ambiguous tile location lies within a relatively dark region of 

30 scanned image 34, or may be lower than 50% is the ambiguous tile location lies 
within a relatively light region of scanned image 34. For example, in one 
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embodiment, 75 % matched filters are generated for each microscreen for relatively 
dark ambiguous tile locations, and 25% matched filters are generated for each 
microscreen for relatively light ambiguous tile locations. The process of FIG. 6 may 
be repeated for one or more different image levels. If a dominant microscreen 

5 cannot be resolved after the process of FIG. 6 has been repeated (step 116), one of 
the microscreens may be selected to be identified with the ambiguous tile location 
and this may be recorded as a "soft" decision in the final score map. The ambiguity 
ultimately may be resolvable by an error correction code, redundancy or other 
decoding feature that has been incorporated into the information embedded in 

10 scanned image 34. 

Referring to FIG. 11, in one embodiment, uncertainties may be resolved as 
follows. Initially, the threshold that was applied to each of the score maps in step 94 
of the process of FIG. 6 may be lowered to identify possible pinnacles at the 
uncertain tile locations (step 120). In general, the threshold should not be lowered 

15 below a certain level to avoid a substantial increase in false positives. If a single 
pinnacle is identifiable using the lowered threshold (steps 122 and 124), the 
corresponding microscreen is identified with the uncertain tile location (step 126). If 
pinnacles for more than one microscreen are identifiable using the lowered threshold 
(steps 122 and 124), the ambiguity may be resolved in accordance with the process 

20 of FIG. 10 (step 126). If no pinnacles are identifiable using the lowered threshold 
(step 122), the uncertainty may be resolvable by an error correction code, 
redundancy or other decoding feature that has been incorporated into the 
information embedded in scanned image 34 (step 128). 

In some embodiments, the number of uncertainties may be reduced by re- 

25 mapping the tone of original image 16 before the halftoning process that is 

implemented by encoder 12 (FIG. 1A). In this way, regions of original image 16 that 
are either too dark or too light for embedded halftone matrices to be detected may be 
removed. The image tone may be re-mapped by a conventional image tone re- 
mapping process. In one embodiment, the image tone may be re-mapped by 

30 clamping the high and low ends of the image level spectrum to respective moderate 
values. For example, for a grayscale image with 255 gray levels, gray levels above 
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240 may be re-mapped to a gray level of 240 and gray levels below 11 may be re- 
mapped to a gray level of 11. In another embodiment, the image tone may be re- 
mapped in accordance with a selected tone curve. 

Referring to FIG. 12, after ambiguities and uncertainties have been resolved in 

5 accordance with the methods of FIGS. 10 and 11, respectively, the identified 
microscreens may be arranged in tile locations across scanned image 102 in an 
ordered sequence of microscreens. In this embodiment, information 14 has been 
embedded in scanned image 102 as an ordered sequence of first and second halftone 
microscreens, which are marked by the differently shaded tile regions 140, 142, 

10 respectively. As shown, in some regions 144, no microscreen could be identified. 
These regions are marked as "missing" by sequence recovery module 72 (FIG. 5). As 
mentioned above, the information contained in missing tile regions still may be 
recovered by subsequent processing of an error correction code, redundancy or other 
feature that has been incorporated into the information embedded in scanned image 

15 34 to facilitate decoding. 

The systems and methods described herein are not limited to any particular 
hardware or software configuration, but rather they may be implemented in any 
computing or processing environment, including in digital electronic circuitry or in 
computer hardware, firmware or software. The encoder and decoder modules may 

20 be implemented, in part, in a computer program product tangibly embodied in a 
machine-readable storage device for execution by a computer processor. In some 
embodiments, the encoder and decoder modules preferably are implemented in a 
high level procedural or object oriented programming language; however, the 
algorithms may be implemented in assembly or machine language, if desired. In any 

25 case, the programming language may be a compiled or interpreted language. The 
encoding and decoding methods described herein may be performed by a computer 
processor executing instructions organized, e.g., into program modules to carry out 
these methods by operating on input data and generating output. Suitable processors 
include, e.g., both general and special purpose microprocessors. Generally, a 

30 processor receives instructions and data from a read-only memory and/or a random 
access memory. Storage devices suitable for tangibly embodying computer program 
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instructions include all forms of non-volatile memory, including, e.g., semiconductor 
memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic 
disks such as internal hard disks and removable disks; magneto-optical disks; and 
CD-ROM. Any of the foregoing technologies may be supplemented by or 
5 incorporated in specially-designed ASICs (application-specific integrated circuits) . 

Other embodiments are within the scope of the claims. 

For example, although the above embodiments are described in connection 
with information embedded as an ordered sequence of individual microscreens, other 
embodiments may embed information in an image as an ordered sequence of groups 
10 (e.g., M x N arrays) of microscreens. 

Still other embodiments are within the scope of the claims. 
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